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PREFACE 


This  expository  RAND  Memorandum  Is  an  outgrowth  of  earlier  work 
written  for  inclusion  in  a  series  sponsored  by  Management  Science  - 
It  was  felt  that  this  somewhat  expanded  version  warranted  publication 
as  a  RAND  Memorandum  because  of  Its  application  to  problems  In  model¬ 
ling,  analyzing,  and  optimizing  systems  that  frequently  arise  In  the 
Air  Force.  The  study  adds  to  a  number  of  other  RAND  Memoranda  that 
deal  either  directly  or  tangentially  with  the  general  subject  of 
semi-Markov  processes  (see  References) . 

As  the  title  Indicates,  this  Memorandum  is  intended  as  an  intro¬ 
duction  to  the  subject  of  semi-Markov  processes.  Only  a  moderate  degree 
of  mathematical  sophistication  Is  required  to  read  It.  The  presenta¬ 
tion  Is  unconventional,  however,  and  this  Is  deliberate.  It  permits 
a  more  heuristic  discussion  of  a  subject  that  has  a  high  intuitive  con¬ 
tent  but  that  may  be  masked  In  an  abstract  setting  Intended  to  rigor¬ 
ously  cover  all  pathological  cases.  Measure- theoretic  niceties  end 
many  other  recondite  matters  are  omitted.  It  Is  expected  that  readers 
primarily  Interested  In  applications  will  be  grateful  for  these  omis¬ 
sions,  but  those  whose  interest  In  semi-Markov  processes  per  se  has  been 
stimulated  will  certainly  wish  to  consult  the  references.  In  addition, 
some  topics  that  arise  In  applications  are  only  briefly  mentioned  along 
with  appropriate  citations  for  the  Interested  reader. 

It  did  not  seem  feasible  to  write  a  primer  for  readers  lacking  any 
knowledge  of  renewal  theory  and  Markov  chains.  Besides,  good  presenta¬ 
tions  of  renewal  theory  arc  already  available  in  Refs.  [1*1  and  [16],  and 
only  a  few  elementary  facts  about  Markov  chains,  which  can  be  found  in 
most  textbooks  on  probability  theory,  have  been  used. 
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SUMMARY 

Marrying  renewal  processes  and  Markov  chains  yields  semi-Markov 
processes;  the  former  are  special  cases  of  the  latter.  This  exposi¬ 
tory  Memorandum  outlines  some  of  the  main  properties  of  the  union. 
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I.  INT80DUCTI0M 


A  key  concept,  already  familiar  to  readers  acquainted  with 
queueing  theory,  la  that  of  an  Imbedded  Markov  chain  connecting  re¬ 
generation  points  of  a  stochastic  process.  Starting  from  a  regenera¬ 
tion  point,  the  future  is  stochastically  Independent  of  the  past, 
given  the  present.  In  the  Imbedded  Markov  chain,  the  original  time 
scale  of  the  transitions  between  regeneration  points  is  replaced  by  a 
discrete  time  version  tihere  all  transitions  take  unit  time.  The  cor¬ 
responding  imbedded  semi-Markov  process  looks  only  at  the  "states" 
corresponding  to  these  regeneration  points,  but  it  does  so  In  continuous 
time. 

The  use  of  these  distinguished  states,  generally  referred  to  as 
events  by  probabllists,  is  natural  In  a  dynamic  progranming  framework 
as  described,  for  example,  in  Denardo  and  Mitten  [13].  In  fact.  It 
was  programming  over  semi-Markov  processes  that  motivated  our  interest 
in  these  processes.  We  shall  have  more  to  say  about  this  in  Sec.  X. 

Part  of  the  definition  of  a  distinguished  state  Is  its  association 
with  certain  regeneration  points.  To  fix  this  Idea  concretely,  con¬ 
sider  the  M/G/1  queue  (Poisson  arrivals,  general  service  time  distribu¬ 
tion,  single  channel).  For  many  purposes,  a  convenient  set  of 
distinguished  states  Is  [0,  1,  2,  ...}  where  state  1  signifies  1  cus¬ 
tomers  In  the  system  and  a  service  has  Just  been  completed. 

This  procedure  has  been  found  helpful  as  a  modeling  device  In 
applications,  although  It  Is  definitely  In  conflict  with  the  notion 
of  a  state  used  in  fully  rigorous  treatments  of  probability  theory 
and  semi-Markov  processes  In  particular.  There,  a  sample  path  of  a 
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stochastlc  process  is  defined  as  a  function  X{»,  ou) :  [0,  <»)-«£  and 
the  members  of  E  are  called  states.  This  often  necessitates  speaking 
of  a  "holding  time"  in  a  state.  If,  as  in  semi-Markov  processes,  this 
holding  time  is  not  necessarily  exponentially  distributed,  then  we 
intuitively  feel  that  the  proces;  does  not  remain  in  the  same  state 
during  the  holding  period  because  the  process  has  memory  (is  not 
Markovian).  Thus,  with  usual  definition  of  state,  two  processes  in 
the  same  state  could  have  different  conditional  future  paths,  depend¬ 
ing  on  their  respective  histories.  By  contrast,  all  our  distinguished 
states  are  occupied  only  for  an  instant,  and  previous  history  is  irrele¬ 
vant,  This  Interpretation  is  unconventional  and  heuristic,  but  the 
underlying  mathematics  remains  the  same.  It  is  intended  to  make  it 
easier  for  those  readers  with  some  prior  exposure  to  Markov  chains  and 
renewal  theory  to  get  a  feel  for  what  semi-Markov  processes  are  all 
about  and  to  learn  to  see  imbedded  semi-Markov  processes  in  applications. 
For  many  readers  belonging  to  this  class,  it  is  felt  that  this  primer 
is  a  more  accessible  first  introduction  than  the  original  papers  cited 
in  the  reference  list. 


Letting  Nj(t)  denote  the  number  of  times  state  J  is  entered  in 
the  half-open  Interval  (0,  t],  we  obtain  the  Markov  Renewal  Process 


(MRP)  N(t)  •  (N^{t),  N^(t),  NjCt)  ,  ...).  In  the  M/G/1  queue,  for 
example,  N^(t)  is  the  number  of  busy  periods  completed  in  (0,  t]. 

Let  Z(t),  the  semi-Markov  process,  be  the  last  distinguished  state 


entered  in  [0,  t1.  In  general,  such  a  last  state  is  not  well  defined, 
but  in  the  applications  there  is  virtually  never  any  difficulty.  See 
Sec.  Ill  for  a  discussion  of  this  point.  In  the  M/G/1  queue,  if  at  the 
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last  service  completion  epoch  In  fO,  tl  there  were  1  customers  In  the 
system,  then  Z(t)  “  1;  however,  at  time  t  there  may  be  more  than  1 
customers  In  the  system  due  to  arrivals  since  the  last  service 
completion. 

Is  defined  to  be  the  set  of  distinguished  states,  assumed 
countable.  The  state  transitions  form  a  Markov  chain  with  transition 


be  modeled  as  SMPs :  for  example,  queueing.  Inventory,  and  maintenance 
problems.  Explicit  recognition  of  the  underlying  SMP  often  stream¬ 
lines  the  analyala.  For  details,  see,  e.g.  ,  Pyke  [44 "5,  Barlow  and 
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Proschan  m,  Fabens  [ISI,  Foley  1181,  ^Inlar  14-81,  Neuts  f37-411, 
Lambotte  159],  and  Simon  161].  These  areas  by  no  means  exhaust  the 
possibilities;  e.g.,  Perrin  and  Sheps  [431  and  Weiss  and  Zelen  [52] 
apply  SMPs  to  medical  problems.  John  McCall  [60]  has  used  SMPs  to 
model  movements  among  income  classes  in  a  study  of  strategies  for 
combatting  poverty.  For  proofs,  citations  of  earlier  papers,  and 
additional  topics  in  SMPs,  the  reader  should  consult  the  reference 
list.  Another  expository  paper  is  Janssen  [26].  Howard  [58]  develops 
a  flow  graph  interpretation  of  SMPs. 

Recently,  Neuts  [42]  has  published  a  bibliography  on  SMPs.  1 
thank  Professor  Neuts  for  bringing  several  relevant  papers  to  my 


attention. 
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11.  DISTIWGUISHED  STATES 

In  this  section  we  examine  precisely  what  is  meant  by  a  distinguished 
state.  Since  discussing  this  topic  in  an  offhand  manner  could  result 
in  confusion,  the  subject  is  treated  in  some  detail. 

For  each  sample  path  of  a  stochastic  process  X,  there  is  a 
correspondence  between  [t:  t  2  0]  and  a  set  of  states  S.  If  every 
state  (in  our  sense  of  the  word)  in  S  is  required  to  have  the  Markov 
property,  then  in  general  S  will  be  uncountable  since  a  history  of  the 
process,  or  at  least  the  relevant  portion  of  it,  must  be  part  of  the 
state  definition.  All  that  we  require  of  S,  however,  is  that  it  have  an 
appropriate  countable  subset  1**^  of  (distinguished)  states  having  the 
Harkov  property.  Thus,  for  a  state  to  be  a  candidate  for  1^,  it  must 
correspond  to  a  regeneration  point,  but  we  do  not  require  that  all 
states  corresponding  to  regeneration  points  belong  to  1^.  We  assume 
that  the  process  starts  in  a  distinguished  state  at  time  C. 

Note  that  the  set  of  distinguished  states  used  previously  in 
our  discussion  of  the  M/G/1  queue  does  not  include  all  regeneration 
points,  since  any  time  the  system  is  idle  (empty),  it  is  at  a  regen¬ 
eration  point.  Occasionally,  it  is  convenient  to  add  to  l'*'  the 
state  corresponding  to  arrival  epochs  to  an  eiapty  system.  Our  choice 
of  distinguished  state*  conforms  to  our  requirements  because  the 
time  to  the  next  arrival  is  stochastically  independent  of  the  time 
elapsed  since  the  last  arrival.  In  general,  arrival  epochs,  except 
those  corresponding  to  the  start  of  a  busy  period,  are  not  regenera¬ 
tion  points.  Thus  the  state  "i  customers  in  the  system  and  a  cus- 
toster  has  Just  arrived"  cannot  be  a  distinguished  state,  unless 
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i  ■  0  or  the  service  times  are  exponential.  Through  the  use  of  so- 
called  supplementary  variables,  we  can  define  every  state  in  the 
original  process  so  that  it  is  Markovian.  Each  state  is  then  a  couple 
of  the  form  (l,u),  which  denotes  1  customers  in  the  system  and  the 
customer  being  processed  has  been  in  service  for  time  u;  for  i  =  0, 
u  is  arbitrary,  say  0.  Sometimes  supplementary  variable  techniques 
are  useful  as  an  alternative  or  adjunct  to  SMP  techniques;  see,  e.g., 

Cox  and  Miller  [9].  A  disadvantage  of  supplementary  variable  techniques 
is  that  superfluous  regularity  conditions,  e.g.,  absolutely  continuous 
transition  time  distributions,  often  must  be  imposed  on  the  original 
process  to  Justify  their  use. 

Returning  to  the  general  discussion,  we  require  the  distinguished 
states  to  be  defined  such  that  nonzero  holding  times  in  a  distinguished 
state  are  forbidden^  but  instantaneous  transitions  among  the  distinguished 
states  are  allowed.  This  is  a  departure  from  the  setup  of  Pyke  [4A], 
although  the  two  formulations  are  essentially  equivalent.  The  notion 
of  an  auxiliary  path  r471,  needed  in  the  conventional  setup  to 
handle  such  processes  as  the  M/C/l  queue,  is  not  required  by  us.  Our 
definition  of  distinguished  state  permits  a  graphic  representation 
of  SMPs  in  terms  of  networks  with  branch  nodes  (distinguished  states) 
and  stochastic  arc  lengths  fl^l.  For  example,  traversing  an  arc 
could  correspond  to  a  customer  completing  service. 

^By  contrast,  in  the  conventional  setup  nonzero  holding  times 
are  essential,  but  this  seems  to  be  an  artifice  unless  at  every 
instant  the  process  is  memoryless,  i.e.,  Markovian. 
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To  remove  ambiguity  in  the  case  of  Instantaneous  transitions,  we 
define  X^(t)  «  X(t^);  thus,  X^  is  right  continuous  and  the  last  dis¬ 
tinguished  state  of  X*^  entered  in  [0,  t1,  say,  is  well  defined, 
provided  that  the  process  does  not  explode;  see  the  discussion  of 
regularity  in  Sec.  III.  Since  we  have  prohibited  nonzero  holding  times 
in  distinguished  states,  we  cannot  allow  a  distinguished  state  to  cor¬ 
respond  to  a  nondegenerate  Interval  of  regeneration  points  (e<g.  ,  an 
idle  period  in  an  M/G/1  queue).  Thus,  to  exclude  an  infinite  sequence 
of  Instantaneous  transitions  from  a  state  to  itself,  we  require  that 
the  distinguished  states  be  defined  such  that,  for  all  nondegenerate 
Intervals  (a,  b)  ,  16  1^  =>  PfX(t)  =  1,  Vt  €  (a,  b)}  =  0.  For  example, 
in  the  M/G/1  queue  it  does  not  suffice  to  define  the  distinguished 
state  0  as  0  customers  in  the  system.  The  condition  that  a  service 
has  Just  been  completed  must  be  added. 

In  applications,  the  first  step  is  to  specify  the  distinguished 
states  carefully,  which  throughout  the  sequel  are  simply  called  "states," 
necessitating  definitions  slightly  different  from  conventional  usage. 

To  make  the  states  correspond  to  regeneration  points  often  requires  a 
detailed  state  description,  calling  for  considerable  disaggregation. 

This  has  the  effect  of  multiplying  naive  estimates  of  the  number  of 
states. 

CAUTION:  The  "state"  of  the  underlying  stochastic  process  and 
the  state  of  the  imbedded  SMP  generally  differ.  Except  at  a  countable 
set  of  (regeneration)  points,  the  state  of  the  underlying  process  is  not 
a  distinguished  state. 

2 

Sometimes  it  is  convenient  to  permit  instantaneous  transitions, 
for  example,  Denardo  [ll].  Yackel  [35]  makes  a  detailed  study  of  limit 
theorems  for  SMPs  with  instantaneous  transitions  allowed. 
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111.  REGULAR  SMPs 

In  the  literature  the  Z  process  Is  called  an  SMP.  But  the  MRP 
and  the  SMP  are  different  aspects  of  the  same  u-iderlylng  stochastic 
process;  therefore,  by  slight  abuse  of  language,  we  sometimes  use  the 
terms  SMP  and  MRP  Interchangeably. 

An  MRP  Is  regular  If  with  probability  one  (w.p.l)  each  state  Is 
entered  only  a  finite  number  of  times  In  any  finite  time  span--l.e., 

If  P[Nj^(t)  <  •]=  1,  VI  €  1^  and  t  a  0.  An  MRP  Is  strongly  regular 
If  w.p.l,  the  total  number  of  state  transitions  Is  finite  in  any  finite 
time  span*-l.e..  If  P[Z  Nj^(t)  <  *1  »  1,  ^  0.  Clearly,  strong 

regularity  Implies  regularity  and.  If  n  <  •,  It  suffices  that 

H  ■  . U^)  have  at  least  one  component  nondegenerate  at  zero 

for  every  ergodlc  subchain  of  the  Imbedded  Markov  chain,  where  Is 
the  unconditional  distribution  of  time  elasped  starting  from  state  1 
until  the  next  state  Is  entered  (possibly  1  Itself).  In  the  denumerable 
state  case  (n  >  <") ,  Pyke  [44]  and  Pyke  and  Schaufele  [461  describe 
conditions  that  Imply  strong  regularity,  as  does  Feller  [171.  In 
the  sequel,  we  assume  that  strong  regularity  holds. 
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In  general,  these  relations  cannot  be  solved  analytically,  although  for 
the  moments,  complicated  expressions  have  been  obtained  (Pyke  and 
Schaufele  1461)  for  irreducible  recurrent  SMPs  (G^^(«d)»  1).  In  the 
finite  state  case,  numerical  solutions  can  be  obtained  by  numerical 
inversion  of  the  corresponding  Laplace  transforms.^  For  each  value 
of  s,  only  one  matrix  inversion  in  the  transform  domain  is  required- 
that  of  I  -  q(s),  where  s  >  0  and 


q(s) 


In  obvious  notation,  having  found  [I  -  q(s)l  either  analytically  as 
a  function  of  s  or  numerically  (for  suitably  spaced  values  of  s),  one 
successively  computes 


m(s) 


I  *  q(s)l'^q(s)  “  ^i  '  qCs)"!*^  -  i 


glj(s)  “  n'^j(s)/[l  +  m^^Cs)! 


1  -  hj(s) 

■  I  -  gjj(s)  ■ 


and  then  inverts  the  transforms.  Although  this  procedure  is  not  trivial, 
it  often  compares  favorably  with  the  alternative  simulation  approach  for 
getting  the  transient  behavior  in  the  time  domain.  By  usual  limit  theo¬ 
rems  for  Laplace  transforms  (Widder  r53'1,  Feller  (161,  see  also  Jewell 
[271),  the  behavior  in  the  time  domain  for  large  (small)  t  corresponds 
to  behavior  in  the  transform  domain  for  small  (large)  s.  Renewal  theory 
provides  an  important  tool  in  studying  asymptotic  behavior;  see  Sec.  VIII 
For  the  stationary  probabilities,  see  Sec.  VI. 

See,  for  example,  Refs.  2,  24,  33,  and  47. 
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Condltlonlng  on  the  event  that  no  state  in  a  subset  B  of  I  Is 
entered  In  (0,  t]  may  be  of  interest.  For  example, 


^  P^^(t)  -  P[Z(t)  -  JIZ(O)  -  1,  Nj^(t)  -  0,  Vk  €  B] 
g  G^j(t)  -  P[N.(t)  >  0|Z(0)  -  1,  Nj^(t)  -  0,  Vk  €  B] 
g  -  E[Nj(t)|Z(0)  -  1.  N^(t)  -  0,  Vk  €  B] 


can  be  calculated  from  the  formulas  already  given  by  (temporarily)  making 
the  states  in  B  absorbing. 

Barlow  and  Proschan  [1,  pp.  132-134],  using  "renewal"  arguments, 

show  that  the  first  and  second  moments  of  ,  denoted  respectively  by 

(2) 

and  are  given  by 

>*1)  •  "ik'kj  •*  'i 


E 


where  is 


the  mean  of  ,  v 


(1) 


v>  and 


vU)  .  r  dH^(t). 

0 

(2)  + 

We  assume  that  '  <  *>  Vi  ^  1  .  If  the  imbedded  Markov  chain  is 
finite  and  ergcdlc,  these  equations  have  a  unique  finite  solution^ 


and, 


with  Tlj 


the  stationary  probability  that  the  last  state  entered 


See  appendices  1  and  2  of  Fox  [19]  for  an  efficient  way  to  solve 
these  equations.  (An  expression  for  the  "bias  terms"  in  Markov  renewal 
progxaasilng  Involves  the  first  passage  time  moments,  which  are  of  in¬ 
trinsic  Interest,  but  recently  Jewell  [291  derived  a  remarkably 
simple  alternative  expression,  obviating  the  need  to  calculate 
these  moments  to  evaluate  the  bias  terms.) 
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is  j  if  all  were  degenerate  at  one, 
by  and  suamlng  yields 


6 


multiplying 


and 


(2) 

ij 


-  (1/tt.) 

k 


*-  kj'j  i  -• 


For  finite  state  SKPs,  the  probability  that  state  J  is  ultimately 
reachmd  starting  from  1  Is^ 


Gij(.) 


1.  if  1.  J  e 

jo.  if  1  €  Ej^.  j  e  E^,  k  #  j, 

C(I  -  if  i  €  T,  J  6 


where  A  Is  the  submetrlx  of  P  corresponding  to  Che  set  T  of  transient 

states,  E, ,  E  are  the  recurrent  subchains  of  P,  and 

’I’m 


e 


I 


1  €  T, 


The  last  case  can  be  checked  using  a  renewal  argument.  A  similar 
argument  shows  that  the  mean  time  Co  leave  T  starting  from  1  la 

-  [(I  -  A)'V3^,  i  €  T, 

where  is  the  vector  of  J  ^  Tyke  [^5]  obtains  a  double 

generating  function  for  the  distribution  of  Mj(t),  viz., 

^In  other  words,  tt  is  the  stationary  measure  for  the  Imbedded 
chain,  but  not  (In  general)  for  the  SMP  Itself. 

^The  case  1,  J  €  T,  of  less  Interest,  la  not  considered. 
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'•'j  i  -(1  ‘  “tz  1  +  (1  -  z)D]‘^  , 

where  _I  Is  a  matrix  of  I's.  D  is  a  diagonal  matrix  whose  positive 
elements  are  the  corresponding  elements  of  (I  -  q)  and 

-  [«^j(z;  .)! 

#ij(z;  s)  -  I  e  d|.w^j(2;  t) 

•b" 

00 

k-0 

v^j(k;  t)  -  PCN^(t)  -  k|z(0)  -  q. 

Thus,  In  principle,  the  probabilities  and  moments  can  be  obtained  In 
the  usual  way.  The  Laplace  transform  m  of  the  first  moment 
was  already  given.  See  Fyke  and  Schaufele  [^b]  for  furtiter  general 
moment  computations,  weak  and  strong  laws  of  large  numbers,  and 
central  limit  theorems.  A  generating  function  that  yields  many 
quantities  of  Interest  upon  considering  special  cases  has  been  ob¬ 
tained  by  Neuts  [36],  Stone  [50]  derives  the  distribution  of  the 
maximum  of  an  SHP. 

Several  tlmea  In  the  sequel  we  use  an  asymptotic  expression  for 
a  transform  valid  for  small  a.  A  good  way  to  check  these  results 
la  to  expand  e  In  a  power  series  and  Interchange  the  order  of  sum¬ 
mation  and  Integration;  this  procedure  can  be  rlgorlzed.  If  a  function 
K  has  Laplace-Stlelt Jes  transform  a/s  b  +  <’(!),  then  K(t)-at  converges 
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in  the  (C,  1),  or  Ces^ro,  sense  to  b.  C 
damp  oscillations,  -.g.,  by  averaging  a 
period.  Under  the  same  conditions  on  K, 
usual  sense  to  a. 


esaro  summation  tends  to 
periodic  function  over  its 
t~^(t)  converges  in  the 


V.  STATE  CLASSIFICATION 


In  classifying  the  states  of  an  SMP  transient,  null  recurrent,  or 
positive  recurrent,  we  must  distinguish  between  a  state's  classifi¬ 
cation  in  the  imbedded  Markov  chain  and  in  the  SMP  itself.  For  l"*^ 
finite  and  <  “>,  Vi  €  1  ,  the  distinction  disappea,-s  and  a  state  j 
is  either  transient  or  positive  recurrent  “  1  and  <  ®) . 

In  large-scale  applications,  the  ergodic  subchain- transient  set 
breakdown  may  not  be  obvious  and  recourse  may  be  necessary  to  an 
algorithmic  classification  scheme  such  as  that  of  Fox  and  Land!  [231. 

For  infinite,  a  state  j  is  transient  (recurrent--!. e. , 

in  the  SMP  «  j  is  transient  (recurrent)  in  the  imbedded  Markov  chain 
State  i  is  positive  recurrent  in  the  imbedded  Markov  chain  (contained 

in  ergodic  subchain  E,  )  and,  for  some  constant  c,  v.  ,  s  c  <  »,  Vi, 

K  1  j 

j  €  =»  /  is  positive  recurrent  in  the  SMP.  An  SMP  is  positive 

recurrent  if  all  tlie  states  in  I^  are  positive  recurrent  in  the  SMP. 

We  remark  that,  if  Z(0)  “  i  and  i  belongs  to  the  same  positive 


recurrent  ergodic  subchain  as  j,  t  Nj(t)  -  l/lAjj  wpl.  a  strong  law 

that  follows  immediately  from  renewal  theory.  Under  these  conditions 

(2) 

and  assuming  <  *,  Nj(t)  is  asymptotically  normally  distributed 

.3  /2)  2 

with  mean  variance  ^  consequence  of  a 

renewal-theoretic  result  found,  for  example,  in  Feller  [16,  p.  359]. 
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VI.  STATIONARY  PROBABILITIES 

It  is  Important  to  distinguish  between  the  stationary  proba¬ 
bilities  with  respect  to  the  imbedded  Markov  chain  and  the 

g 

stationary  probabilities  with  respect  to  the  SMP.  Thus 

Is  the  steady  state  probability  that  the  last  distinguished  state 
entered  is  i.  Hence  the  (p^}  are  of  direct  interest  in  applications, 
while  the  fn^}  are  computed  only  as  an  intermediate  step.  We  consider 
first  the  case  l"*^  finite  and  <  *.  Vi  €  l"^. 

vj/lijj.  J€E^.  Z(0)€E^ 

G.  j  €  Z(0)  -  i  e  T 

0,  j  €  T 

0,  j  €  E^.  Z(0)  €  E^,  k  i 


ZilL 


T,  n 


with  ftr^)  here  being  the  stationary  probabilities  for  the  imbedded 
Markov  chain  given  that  Z(0)  €  Ej^.  note  that  the  expression 
is  intuitively  reasonable  and  follows  from  the  fact  that  " 

^  easily  verified  using  the  expression  for  given 

in  Sec.  IV,  and  a  Tauberian  argument. 


O 

^n  general,  the  stationary  probabilities  must  be  interpreted 
as  Cesaro  limits.  If  the  process  is  aperiodic,  these  reduce  to 
ordinary  limits. 
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In  the  remainder  of  this  section  we  assume  that  the  Imbedded 
Markov  chain  Is  Irreducible  and  that  the  SMP  Is  positive  recurrent, 
where  may  be  finite  or  infinite.  We  also  assume  that  the  mean 
transition  times  are  uniformly  bounded  away  from  zero;  i.e.,  0  <  e  s 
<  *•  With  these  assumptions,  Fabens  flS]  shows  that 


L"t''i 


in  agreement  with  results  given  above  for  the  finite 
as  the  time  of  the  last  transition  completion  before  or 
the  time  of  the  next  transiton  completion  after  x.  The 


case.  Define  ct(x) 
at  X  and  t(x)  as 
random  variables 


V(x)  =  t(x)  -  X  (excess  r  v.) 
6(x)  «  X  -  o(x)  (shortage  r.v.) 


are  of  interest.  Adding  to  the  previous  assumptions  the  hypothesis 

that  Z(')  is  aperiodic,  Fabena  shows  that 


llm  Pf6(t)  S  x|z(t)  -  i)  •  Urn  P{Y(t)  s  x|z(t)  -  1} 
t-«  t.-a> 


[1 


(u) ^  du . 


This  generalizes  the  well-known  result  from  renewal  theory  for  the 
one-state  case,  obtained  there  as  a  corollary  to  the  key  renewal 
theorem  [1]. 
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The  general  question  of  existence  and  uniqueness  of  stationary 
measures  Is  dealt  with  in  Pyke  and  Schaufele  [47].  Cheong  [3]  and 
Teugels  [51]  give  conditions  under  which  convergency  to  the  steady 
state  Is  geometric.  An  estimate  of  the  convergence  ate  Is  Impor¬ 
tant.  If  It  Is  high  enough,  troublesome  transient  phenomena  can  be 
neglected.  We  then  pass  directly  to  a  relatively  simple  steady-state 
analysis . 
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VII.  EXAMPLE:  THE  M/C/ I  QUEUE 

To  illustrate  the  notion  of  stationary  probabilities  for  an  SMP, 

9 

we  consider  the  M/G/1  queue.  Let 

n.  =  the  stationary  probability  that  i  customers  are  in 
the  system  just  after  a  random  service  completion 
epoch 

p.  =  the  stationary  probability  that  i  customers  are  in 
the  system  just  after  the  service  completion  epoch 
preceding  a  random  point  in  time 

p.  =  the  stationary  probability  that  i  customers  are  in 
the  system  at  a  random  point  in  time 

We  assume  that  the  traffic  intensity  ib  is  less  than  one,  where  \  is 

the  arrival  rate  and  b  is  the  mean  service  time,  assumed  positive. 

Although  it  is  easily  shown  from  a  result  in  Si..  VI  and  the  fact  that 

V,  “b,  i>l,y  *l/v+b,  and  -  =  I  -  >b  that 

L  o  o 


Pi 


XbTTj^,  i  i  1 

(1  +  Xb)n^  -  1  -  (Xb)^,  i  -  0, 


it  turns  out  that  p^  “  ’^i’  ^  remarkable  result  originally  due  to 

Khintchine  and  derived  In  a  more  elementary  manner  by  Fox  and 

Miller  [29''  using  SMP  theory.  In  bulk  queues  Tls"',  for  example, 
the  stationary  measures  for  the  imbedded  Markov  chain  and  the 
original  queueing  process  are  different. 

Readers  familiar  wi'.h  queueing  theory  may  prefer  to  skip  to  the 
last  paragraph  of  this  section.  In  between,  the  standard  manipulations 
yielding  G  (z),  the  generating  function  of  the  I-.),  are  peiformed. 

TT  J.  ' 

- 

The  stationary  probabilities  for  tlie  imbedded  chain  and  the  SMP 
are  and  respectively. 
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Recalling  that  state  n  means  that  there  are  n  people  in  the 
system  and  a  service  has  just  been  completed,  we  obtain  the  well- 
known  transition  matrix  for  the  imbedded  Markov  chain! 


0 

1 

2 

3 

4 

•  •  • 

c 

k, 

A 

“2 

nr 
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• 

where  the  probability  that  n  customers  arrive  wnile  a  customer  is  being 
serviced  is 


k 

n 


(Xt)"  dB(t), 


and  B  is  the  service  distribution.  By  the  usual  straightforward 
manipulations,  we  find  that  the  generating  function  of  the  is 


Gj^(z)  ■  e(X(l  -  2)), 

i 

where  g  Is  the  Laplace-Stlelt Jes  transform  of  B,  i.e., 


0(8) 


dB(t). 


To  obtain  the  stationary  vector  tt  for  the  chain,  we  multiply  the  i-th 
relation  determined  by  rrP  ■  n  by  and  sum,  define  the  generating 
funct  ’■  on 


and  obtain  the  standard  result 


“  Gj^(2)  -  z 


from  the  special  form  of  P  for  this  chain  by  an  easy  calculation. 

Using  the  fact  that  lim  G  (z)  =  1  (i.e.,  tlie  probabilities  sum  to  1) 
z-*l"  ^ 


and  applying  L'Hospital's  rule. 


rr^  -  1  -  Xb. 


(Intuitively,  this  corresponds  to  the  fraction  of  time  the  server 
is  busy  equalling  the  mean  service  time  divided  by  the  mean  arrival 
spacing. ) 

Sunmarlzing  our  results  so  far. 


n^(l  -  z)0(X(l  -  z)) 
B(X(1  -  *))  -  z 


Thus,  using  L'Hospital's  rule,  the  mean  number  In  the  system  averaged  over 

2 

service  completion  epochs  is,  with  a  ihe  variance  of  the  service  times. 


11m  G  '(z)  ■  n  lim 
.  -  TT  o  ,  _ 

z-»l  z-*l 


g;;(z)  -  2C'(«)(C^(z)  -  1) 


2(G^(z)  -  1)‘ 


2(1  -  Xb) 


+  Xb, 
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and  because  Vi  is  also  the  mean  number  in  the  system  at  a 

random  point  in  time  (in  the  steady  state).  Higher  moments  and 
probabilities  can  be  obtained  from  the  generating  function  by  appro¬ 
priate  differentiations,  which,  however,  become  quite  tedious. 

Having  found  G^(2) ,  the  Laplace-Stielt jes  transform  of  the  sta¬ 
tionary  waiting  distribution  (Pollaczek-Khintchine  formula)  for  the 
first-come,  first-served  (FIFO)  discipline  can  be  easily  found.  The 
derivation  depends  on  the  fact  that,  since  the  arrival  process  is 
Poisson,  an  arrival  plays  the  role  of  a  random  observer.  If  an  arrival 
finds  the  system  empty,  the  conditional  wait  in  queue  is  0.  Otherwise, 
it  is  governed  by  the  remaining  processing  time  of  the  customer  in 
service,  the  excess  random  variable  (Sec.  VI),  plus  the  service  times 
for  the  customers  (if  any)  already  in  queue.  Noting  that  ^^(z)  =  G^(z), 
the  interested  reader  can  readily  derive  a  version  of  the  Pollaczek- 
Khintchine  formula,  namely,  rt  +  --  TG  (b(s))  -  rt  I-  See  Feller 

ri6,  p.  3921  for  an  alternate  elegant  derivation  that  bypasses  the 

calculation  of  G  (z).  A  third  derivation  follows  from  the  fact  that 
P 

the  number  of  customers  in  the  system  just  after  a  departure  is  the 
number  of  arrivals  during  his  total  wait  (queueing  time  plus  service 
time);  the  resulting  equation  is  solved  by  taking  generating  functions 
yielding  the  standard  form  of  the  Laplace-Stielt jes  transform  of  the 
stationary  queueing  delay  distribution  sn^/fs  -  X[1  -  BCs)]},  the  more 
familiar  version  of  the  Pollaczek-Khintchine  formula. A  fourth 

^^Comparison  of  the  two  versions  yields  an  Interesting  and 
surprising  identity. 
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derlvatlon,  where  the  (superfluous)  assumption  of  an  absolutely 
continuous  service  distribution  Is  tacitly  made,  is  given  In  Cox 
and  Miller  [9,  pp,  241-242]. 
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VIII.  ASYMPTOTIC  FORM  OF 


From  a  previous  section,  we  know  that 


m.j(s)  -  gj^j(s)[l  + 


gij(s) 


Formally  expanding  e  In  a  Taylor  series  under  the  Integral  defining 
g_,  integrating  termwise,  and  performing  the  appropriate  algebraic 
manipulations  yields  for  1,  J  in  the  same  ergodlc  subchain 


(2) 

.  iii 

‘“JJ  “IJ 


+  0(1), 


whence  by  a  Tauberlan  argument 


(2) 

„  (t)  -  ^ _ ^ 

Ij  (Cesaro)  ^^^^2 


a  result  that  can  be  obtained  by  analogy  with  renewal  theory  for 
delayed  recurrent  events,  where  the  time  to  the  first  "renewal"  has 
distribution  and  the  spacing  between  subsequent  renewals  has 
distribution  If  the  SMP  is  aperiodic,  the  Cesaro  limit  reduces 

to  an  ordinary  limit.  It  can  be  shorn  that  the  formal  manipulation 
used  CO  obtain  the  asymptotic  expansion  of  m^j(s)  is  Justified  if 


(2)  .  _  ....  . . (2)  ^  _  w,  .  T+  ,  (2) 


Hjj  <  *.  If  1  is  finite,  <  »,  Vi  €  I 


'JJ 


ee.  A  result 
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that  holds  for 
from.  Barlow  and 
inequality  from 
bound  as  well. 


Ill  t  is  2(0)  -  j  ,M..(t)  ^  t/^.,  -  1.  Which  follows 
Proschan  [1.  Theorem  2.5].  We  can  obtain  a  tighter 
[1]  and.  if  has  increasing  failure  rate,  an  upper 
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DC.  FINITE  SHPs  WITH  COSTS 


Sometimes  system  performance  is  evaluated  by  probabilistic 

critfria,  smli  as  average  delay  in  a  queue,  that  serve  as  surrogates  for 

monetary  loss.  It  is  more  appealing  to  deal  directly  with  expected  loss 

11 

as  a  performance  measure.  In  this  section  we  indicate  how  to  do  this. 

Often  in  applications,  costs  are  associated  with  the  transitions. 
Measuring  time  from  the  start  of  an  i  -•  j  transition,  let  C^^(x|t)  be 
the  cost  incurred  up  to  time  x  given  that  the  transition  length  is  t. 

The  expected  discounted  cost  for  a  transition  starting  from  state  i 
is  then 


/m  t 

dFi.(t)  f  d^C.j(xlt). 

V  u  •No¬ 


where  a  cost  incurred  at  time  x  is  discounted  by  the  factor  e  . 

An  elementary  renewal  type  argument  then  shows  that  v^(cr),  the  total 
expected  discounted  cost  over  an  infinite  horizon  starting  from  state 


i,  satisfies 


v(of)  “  V(a)  +  q(a)v(Qf), 


where  o  >  0  and  v(->)  and  yCa)  are  vectors  with  i-th  components  \ 
and  \^(a),  respectively.  Thus,  assuming  a  finite  number  of  states, 


v(of)  “  [1  -  qCa)]  y(a) 


A  similar  remark  applies  to  "chance-constrained"  programming. 
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Since  I  -  q(0)  is  singular  and  a  direct  asymptotic  expansion  is  not 
obvious,  it  is  convenient  to  use  the  relation  between  q  and 
m  given  earlier  to  study  the  behavior  of  v(a)  as  or  —  o"*^.  Following 
Jewell  [281,  we  liave 

v(a)  =  [l  +  mCof)  ]Y(o) 

Making  use  of  the  expansion  of  m(of)  given  in  the  preceding  section, 
we  find  that,  if  i  is  a  recurrent  state,  v^(a)  has  the  form 

Vj^(a)  •=  L.^/tg  +  Wj^  +  o(l) 

and  a  straightforward  argument  in  Fox  [19]  then  shows  that  this  form 
is  valid  for  any  state;  i.e., 

v(«)  ■  i/of  +  w  +  o(l)  . 

Expressions  for  t,  the  loss  rate  vector,  and  w,  the  bias  term  vector, 
can  be  found  In  Jewell  [28,  29"'  and  Fox  119"',  where  appropriate 
conditions  are  given  to  justify  the  expansion.  Substituting  this 

relation  into  v((y)  =  +  q(ar)v(ar)  and  equating  the  coefficients 

- 1  12 
of  o  and  the  constant  terms,  respectively,  yields 


^^his 


procedure  can  be 


justified  by  a  simple  contradiction 


argument.  Note  that  qij(a)  ■  Pij(l  -  OVij)  +  0(0)  and  that  the 
loss  rate  for  all  states  in  an  ergodic  subchain  is  the  same. 
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These  expressions  can  be  solved  uniquely  for  I,  but  w  is  determined 
only  up  to  an  additive  constant  in  each  ergodlc  subchain,  as  described  In 
Denardo  and  Fox  [12]-  An  interesting  and  intuitive  result  that  follows 
easily  from  the  first  formula  is  that  the  loss  rate  for  each  state  In 

an  ergodic  subchain  is  the  same  and  equal  to  n'  wliere 

fk) 

T!  '  is  the  stationary  vector  for  the  corresponding  submatrix  and 
and  are  the  restrictions  Y  and  v,  respectively,  to  This  formula 

can  be  rewritten  as  (cf.  Sec.  VI) 

which  is  the  sum  of  the  expected  cost  per  unit  time  in  each  state  of 
weighted  by  tlie  respective  stationary  probabilities  for  the  SMP. 

Having  first  computed  the  loss  rates  for  the  recurrent  states,  the 
loss  rates  for  the  transient  states  are  obtained  from  the  fact  re¬ 
flected  in  Pjl  “  I  that  the  loss  rate  for  a  state  is  given  by  the  ap¬ 
propriate  convex  combination  and  that  I  -  A  is  invertible,  where  A 
is  the  submatrix  corresponding  to  the  transient  states. 

Denoting  tlie  undiscounted  loss  up  to  time  t  by  L(t),  we  obtain 
from  the  asymptotic  expansion  of  v(a) 

L(t)  -  ft  - ►  w. 

(Cesaro) 

Jewell  r30]  studies  the  fluctuations  in  cumulative  loss  in  what 
is  essentially  the  one-state  case.  If  the  imbedded  Markov  chain  is 
ergodic,  these  results  extend  in  principle  to  n-state  problems  by 
considering  and  tlie  distribution  of  cumulative  loss  until  the 
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flrst  return.  In  general,  the  calculation  would  be  tedious.  Besides, 
we  are  distinguishing  here  only  one  state  out  of  n.  Apparently,  no 
one  has  dealt  with  fluctuation  theory  for  the  n-state  case  directly. 

A  related  topic  Is  a  central  limit  theorem  for  cumulative  loss. 
Since  cumulative  loss  is  an  example  of  a  functional  of  a  Markov 
Renewal  Process,  results  of  Pyke  and  Schaufele  [^61  apply. 


MARKOV  RENEWAL  PROGRAMMING 


The  slCuaCion  becomes  more  interesting  when,  at  each  state  i,  one 
has  a  set  of  options  A^^  and  the  choice  at  1  simultaneously  determines 
P.  F.  and  C.  .  for  all  i  €  l"*".  The  goal  is  to  choose  a  policy  that 

ij  ij’  ij  °  tr  j 

minimizes  either  the  expected  discounted  loss  or  the  loss  rate.  In 

the  latter  case,  an  appropriate  secondary  objective  is  to  minimize 
(i ,  w)  lexicographically,  which  is  especially  important  when  some 
policies  can  have  transient  states.  With  either  criterion,  an  optimal 
policy  can  be  found  by  linear  programming  when  and  ^  j;gl+ 
are  finite.  Details  may  be  found,  for  example,  in  Jewell  [28,  291,  jox 
[19,  221,  Denardo  [10,  111,  and  Denardo  and  Fox  [121,  where  references 
to  the  earlier  (extensive)  literature  on  the  subject  are  given.  The 
linear  programming  formulation  facilitates  sensitivity  analyses  and 
parametric  studies.  Controlling  roundoff  errors  is  probably  less 
difficult  in  the  averaging  version,  as  indicated  in  [12]. 

Some  papers  treat  the  l'*'  infinite  case,  but  the  author  believes 
that,  for  applications,  the  general  theory  developed  so  far  for  that 
case  is  inadequate,  since  "policy  evaluation"  requires  solving  an  in¬ 
finite  system  of  equations,  and  that  particular  problems  are  best  at¬ 
tacked  on  an  ad  hoc  basis.  The  averaging  version  of  the  Infinite  state 
case  apparently  has  been  studied  only  in  the  discrete  time  setup, 
described,  for  example,  in  Derman  [14]  anJ  Ross  [48],  In  Che  discounted 
contlnous  time  version,  however,  no  new  theoretical  problems  arise  when 
the  problem  is  approached  via  contraction  coappings  [lOl.  Fox  [20] 
describes  the  case  where  1^  is  finite  but  the  finiteness  restriction  on 


Xa^  is  dropped.  The  connection  with  generalized  linear  programming  and 
column  generators  is  outlined  in  [12']. 

Miller  [32,  BS"!  treats  the  continuous  time  Markov  process  case 
with  loss  proportional  to  the  transition  time.  His  results  go  beyond 
what  has  been  obtained  for  general  Markov  Renewal  Programs. 

Markov  renewal  programming  problems  are  a  fertile  source  of  large- 
scale  linear  programs.  Many  problems  that  look  deceptively  simple  at 
first  can  lead  to  linear  programs  with  hundreds  or  thousands  of 
constraints  because  of  the  detailed  state  description  required  to 
make  all  decision  points  regeneration  points.  But  often  we  need  not 
throw  ill  the  towel.  Anuthei  luuk  generally  reveals  that  the  constraint 
matrix  is  sparse  and  structured  so  as  to  be  amenable  to  decomposition. 
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XI.  ESTIMATION.  IMFERENCE.  AND  ADAPTIVE  CONTROL 

Moore  and  Pyke  r351  develop  estimators  for  the  ^nd  the 

and  their  large  sample  distributions.  And  Wolff  [54]  describes 
statistical  inference  in  birth  and  death  queueing  models.  Both  of  the 
foregoing  approaches  are  objectivist,  l.e.,  non-Bayeslan,  When  a  large 
number  of  observations  are  at  hand,  the  objectivist  approach  is  un¬ 
objectionable  and  difficulties  stemoiing  from  a  possible  lack  of  con¬ 
sensus  of  prior  belief  do  not  emerge.  On  the  other  hand,  when  the 

observations  are  few  or  nonexistent,  as  is  common,  a  Bayesian  approach 

13 

incorporating  prior  beliefs  and  loss  functions  is  essential.  Such 
an  approach  may  be  formal  or  may  simply  consist  of  a  sensitivity  analy¬ 
sis  with  the  outcomes  being  given  subjective  weights.  In  the  realm  of 
decision  making,  policies  should  adapt  to  modified  beliefs  as  more 
observations  are  taken. 

Unfortunately,  when  the  tradeoff  between  information  acquisition 
and  immediate  losses  is  explicitly  included  in  the  problem  formulation, 
the  number  of  states  generally  explodes.  Generally,  explicit  inclusion 
is  advisable  because,  if  an  average  cost  criterion  is  interpreted 
literally,  policies  that  are  absurd  for  any  positive  discount  rate  can 
result.  Thus,  from  a  practical  viewpoint,  "optimal"  adaptive 
policies  can  be  found  for  relatively  few  problems  [21];  for  the 
remainder,  it  appears  that  we  must  be  content  with  heuristic  devices. 
This  area  remains  largely  unexplored  and  is  ripe  for  investigation. 


^\hi8  is,  of  course,  a  statement  of  the  author's  opinloii.  These 
matters  are  highly  controversial. 
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